Manifold Learning with Local Linear Embedding

Several techniques approximate a lower dimensional manifold. One example is locally-linear embedding (LLE) that was developed in 2000 by Sam Roweis and Lawrence Saul.

This notebook demonstrates how LLE ‘unrolls’ the swiss roll, and how it performs on other data.

For each data point, LLE identifies a given number of nearest neighbors and computes weights that represent each point as a linear combination of its neighbors. It finds a lower-dimensional embedding by linearly projecting each neighborhood on global internal coordinates on the lower-dimensional manifold and can be thought of as a sequence of PCA applications.

Imports & Settings

A Simple Linear Manifold: Ellipse in 3D

PCA: Linear Dimensionality Reduction finds some manifolds

PCA is able to recover the 2D manifold.

Swiss Roll

Create 2D version by sorting datapoints by value and plotting using x, y coordinates only

The left box shows the 3D version, the right box shows the 2D manifold that is home to the swiss roll.

Linear cuts along the axes

Linear methods will have a hard time.

Can PCA find the Swiss Roll Manifold?

PCA is unable to capture the manifold structure and instead squashes the swiss roll sideways:

Manifold learning can make a classification task linear

We'll compare two cases with very different spatial location of the classes.

In this case, the classes are linearly seperable given their manifold.

Local-Linear Embedding learns the Swill Roll manifold

S-Curve

Can PCA identify the S-Curve Manifold?

Local-Linear Embedding

Handwritten Digits - MNIST Data

Load Data

Visualize Data

PCA

Explained Variance

The below plot illustrates how much more variance PCA is able to capture for individual digits.

2D Projection

Local Linear Embedding

The following locally_linear_embedding on mnist.data takes fairly long to run, hence we are providing pre-computed results so you can explore the visualizations regardless of your hardware setup.

2D Projection

Plotly Visualization

Fashion MNIST Data

Load Data

Visualize Data

PCA

Explained Variance

The below plot illustrates how much more variance PCA is able to capture for individual digits.

2D Projection

3D Projection

Local Linear Embedding

2D Projection

Plotly Visualization